For years, researchers have been struggling to bring immersive experiences to home entertainment. Based on the observation that larger display with a wider field of view may cause the user more immersed and present-in experience, “Focus+Context” display was first proposed to have a high-resolution display (focus) surrounded by a lower resolution projection screen (context) to enhance immersive experiences. This way, users perceive the scene-consistent, low-resolution color, light, and movement patterns projected into their peripheral vision as a seamless extension of the primary content.
As an extension of the “Focus+Context” concept, the Infinity-By-Nine project displayed video content on three projector screens surrounding a television. The system uses optical flow, color analysis, and pattern-aware out-painting algorithms to create a synthetic light field beyond the screen edge and projects it onto walls, ceiling, or other suitable surfaces within the peripheral awareness of the viewer.
Recently, IllumiRoom project of Microsoft Research advances this idea to high-resolution projected content, which is dynamically adapted to the TV program content and user interaction. IllumiRoom does not require installing expensive projection screens, nor changing anything about the existing living room, which leverages the existing physical environment to create unique virtual-physical game experiences.
However, the IllumiRoom system is aiming at providing users a more magical gaming experience, and all illusions that it can create need the triggers from the specially-designed games, which limits its usability to the people who are not game fans. A more general framework to provide user immersive home entertainment is expected to get viewer more engaged into the program with more supporting content with lighting/illusion effects.
Nowadays, mobile devices, such as smartphones and tablets, are becoming more powerful in terms of computing and display, projectors are becoming smaller, quieter, and clearer. TVs are becoming a big screen computer with easy access to the Internet and many video content providers (e.g., YouTube and Netflix), and link capacity around individual device has been significantly increased. The home entertainment systems are experiencing revolutionary changes. Many devices surround people in their daily lives, and provide various ways to access, retrieve, and view contents. People use different devices at different scenarios. For example, a user may enjoy a show on a big-screen TV at home, or on a tablet when going to a room without TV, and continue the experience on a smartphone when stepping out of the house. Such continuous and seamless viewing experience requires collaboration among devices and intelligence in determining user intension.
On the other hand, according to Simply Measured, the in-depth measurement across popular online social networks (OSNs), including Facebook, video is shared 1200% more times than links and text combined. Video streaming is tightly coupled with social interaction. Links to popular videos can be quickly spread through OSNs. Videos can also generate social interactions. In addition, several studies show a clear tendency of the user to use tablet or smartphone when watching television, and other studies distinguish a higher percentage of comments or posts on social networks, about the content being watched. Sidecastr is a TV companion app for scene-synchronized enjoyment of high quality social comments tweeted by fans, friends, and celebrities. This system advances the idea to use intelligence to deliver better experience to users, such as the audio detection, and seamlessly keeps comments in constant sync with whatever the users are watching on the primary TV screen. It provides users the manually filtered information of the live Twitter stream without the spam, off-timed comments. Moreover, people search related information and interact with people in different forms on smart devices when watch TV, such interaction helps to direct users to content that suit their interests and needs. Hence, enhanced entertainment experience requires collaboration among people.
AmigoTV proposed the concept to let viewers communicate during watching broadcast. People are represented by their selected avatar in overlay on their TV screen and that of their buddies that are watching the same channel. But all graphics and avatars of the friends are shown in overlay on the broadcast channel, claiming that this is the way to immerse the viewer into the content. However, due to the limit of the screen size and the property of the program, users are likely to complain about the content to be covered and feel the social information annoying.
From the technical point-of-view, to increase the overall link capacity around individual devices, collaborative systems have been proposed in Microcast, in which short-range links such as WiFi and Bluetooth are utilized. Although Microcast achieves good streaming rates by utilizing both 3G and WiFi channels, it is limited by the centralized scheduler and is challenged by energy consumed by the complex coding operations. Mining OSNs may uncover useful information on sharing patterns and hints on efficient delivery of streaming content. For this reason, AMES-Cloud, a framework for mobile multimedia streaming, is proposed, in which video content is pre-fetched according to online social activities. Similarly, MBoard considers social links to direct a multimedia sharing system for online forums.
Given the advancements in hardware and in devising information from OSNs, the primary challenges for providing such a collaborative entertainment system are integrating of devices, compiling useful information according to the needs of users, and providing non-distracting methods for social interaction. From the human computer interaction perspective, the challenge is how to transform the passive lonesome TV consumption behavior into a compelling social experience. From the technology perspective, the challenges are aggregating information from different services, synchronization of content delivered over different networks, and rendering and displaying information and content according to preference of users. The information presentation scalability requires the system has sufficient intelligence to determine the information priorities and fitness before project them onto a view window.
Overall, immersive entertainment is expected to be a collaborative effort among devices, people and content. The disclosed method and system are directed to solve one or more problems set forth above and other problems. It should be noted that, unless explicitly acknowledged, the above background information is part of the present disclosure and is not intended to be prior art.